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In  July  2006 ,  the  309th  Software  Maintenance  Group  (309th  SMXG)  at  Hill  Mir  Force  Base,  Utah  was  appraised  at  a 
Capability  Maturity  Model  Integration  (CMMFM)  Level  5.  One  focus  project  had  been  using  the  Team  Software  ProcessSM 
(TSPfM  since  2001 .  TSP  is  generally  considered  a  Level  5  process;  however ;  during  the  preparation  for  the  assessment ,  it 
became  obvious  to  the  team  that  even  the  stringent  process  and  data  analysis  requirements  of  the  TSP  did  not  completely 
address  CMMI  requirements  for  several  process  areas  (PAs).  The  TSP  team  successfully  addressed  these  issues  by  adapting 
their  process  scripts ,  measures ■  and forms  in  ways  that  may  be  applicable  to  other  TSP  teams. 


In  July  2006,  the  309th  SMXG  was 
appraised  at  a  CMMI  Level  5.  One  of 
the  309th’s  focus  projects,  the  Ground 
Theater  Air  Control  System  (GTACS) 
project,  had  been  using  the  TSP  since 
2001.  The  team  had  achieved  a  four- fold 
increase  in  productivity  during  that  time, 
had  released  zero  defects  since  the  TSP 
was  adopted,  and  had  been  internally 
assessed  at  a  high  maturity  by  the  group’s 
quality  assurance  team.  GTACS  team 
members  felt  confident  they  could  meet 
the  rigors  of  a  CMMI  assessment  and 
achieve  their  group’s  goal  of  Level  5. 

Watts  Humphrey,  who  is  widely 
acknowledged  as  the  founder  of  the 
Capability  Maturity  Model®  (CMM®) 
approach  to  improvement  and  who  later 
created  the  Personal  Software  Process 
(PSP)SM  and  TSP,  has  noted  that  one  of 
the  intents  of  PSP  and  TSP  is  to  be  an 
operational  process  enactment  of  CMM 
Level  5  processes  at  the  personal  and  pro¬ 
ject  levels  respectively  [1].  CMM  and  later 
the  CMMI  were  always  meant  to  provide  a 
description  of  the  contents  of  a  mature 
process,  leaving  the  implementer  with  the 
task  of  definition  and  enactment  of  these 
mature  processes.  Thus,  CMM  and  CMMI 
are  descriptive  not  prescriptive  models. 
The  TSP  goal  of  being  an  operational 
Level  5  process  implies  that  a  team  prac¬ 
ticing  TSP  out-of-the-box  should  be  very 
close  to  being  Level  5. 

The  309th  is  a  large  organization  of 
nearly  800  employees,  both  civil  service 
and  contactors.  The  group  level  is  com¬ 


prised  of  five  squadrons,  each  with  a  dif¬ 
ferent  focus  or  product  line.  309th  man¬ 
agement  and  Software  Engineering 
Process  Group  (SEPG)  sets  group  policy 
and  defines  a  group  level  process  and 
metrics  framework.  Each  squadron 
applies  the  group  level  process  to  its 
technical  domain.  So  projects,  like 
GTACS,  must  ensure  their  detailed  pro¬ 
ject  processes  are  consistent  with  their 
squadron’s  process  and  with  group-level 
guidance.  The  GTACS  project  is  also 
divided  into  several  sub-teams,  all  man¬ 
aged  as  one  project.  The  GTACS  soft¬ 
ware  team,  which  performs  most  of  the 
GTACS  assigned  technical  efforts,  uses 
TSP  to  support  its  work.  A  separate 
Configuration  Management  (CM)  team 
provides  CM  services.  The  project’s  cus¬ 
tomer,  the  GTACS  Program  Office, 
retains  systems  engineering  responsibility 
and  authority.  This  diverse  organizational 
structure  is  important  because  several  of 
the  CMMI  issues  that  need  to  be 
addressed  are  clearly  the  responsibility  of 
these  other  entities  and  were  not  GTACS 
TSP  team  issues  other  than  alignment 
and  coordination. 

Assessment  Timeline 

In  order  to  prepare  for  the  assessment, 
309  SMXG  conducted  a  series  of 
Standard  CMMI  Appraisal  Method  for 
Process  Improvement  (SCAMPISM) 
assessments  which  included  the  GTACS 
team.  There  are  three  kinds  of  SCAMPI 
assessments:  A,  B,  and  C.  The  SCAMPI 


A  assessment  is  the  final  review  during 
which  a  CMMI  level  can  be  determined. 
SCAMPI  Bs  and  Cs  are  less  rigorous  and 
are  intended  to  prepare  the  team  for  the 
full  SCAMPI  A.  The  309th  SMXG  used 
SCAMPI  Bs  to  ensure  compliance  to  the 
model  and  value  added  to  the  enterprise. 
In  general  the  SCAMPI  B  teams  were 
told  to  aggressively  identify  risks  to  a 
successful  SCAMPI  A  appraisal.  When 
the  SCAMPI  B  teams  identified  a 
process  weakness,  they  assigned  a  high, 
medium,  or  low  risk  rating  based  on  the 
seriousness  of  the  noted  weakness. 

From  the  perspective  of  the  TSP 
team  there  were  four  types  of  weakness¬ 
es:  non-team ,  process ,  artifact ,  and  document. 
The  non-team  weaknesses  were  those  that 
were  the  responsibility  of  a  team  other 
than  the  TSP  team,  such  as  the  group’s 
SEPG  or  the  GTACS  CM  team. 
Examples  include  policy  changes  or 
changes  to  the  CM  process.  Process  weak¬ 
nesses  indicate  that  the  team  had  no 
process  in  place.  An  artifact  weakness 
meant  the  assessment  team  found  insuf¬ 
ficient  artifacts  to  pass  the  assessment.  A 
document  weakness  meant  the  team’s 
process  documentation  needed  to  be 
updated. 

The  initial  SCAMPI  B  for  the 
GTACS  focus  project  was  held  about 
one  year  before  the  SCAMPI  A  final 
assessment  and  identified  86  weaknesses. 
A  summary  of  the  counts  and  types  of 
these  weaknesses  is  found  in  Table  1. 
Not  all  weaknesses  were  project  focused. 
Some  were  organizational  and  some 
were  squadron  focused.  Of  the  project- 
focused  risks,  many  were  the  responsibil¬ 
ity  of  one  of  the  following:  overarching 
project  management  (e.g.,  data  manage- 


SM  Team  Software  Process,  Personal  Software  Process,  PSP, 
TSP,  and  SCAMPI  are  service  marks  of  Carnegie  Mellon 
University. 

®  Capability  Maturity  Model  and  CMM  are  registered  in  the 
U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon 
University. 


Table  1:  SCAMPI  B1  Noted  Weaknesses 


Risk  Level 

Total  Risks 

Process 

Risks 

Artifact 

Risks 

Document 

Risks 

Non-Team 

Risks 

High 

19 

1 

17 

0 

1 

Medium 

67 

15 

18 

6 

28 

Low* 

0 

0 

0 

0 

0 

Total 

86 

16 

35 

6 

29 

*  Low  risks  were  not  categorized  in  the  first  SCAMPI  B 
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Risk  Level 

Total 

Risks 

Process 

Risks 

Artifact 

Risks 

Document 

Risks 

Non-Team 

Risks 

High 

23 

0 

8 

0 

15 

Medium 

38 

7 

6 

17 

8 

Low 

22 

0 

1 

11 

10 

Total 

83 

7 

15 

28 

33 

Table  2:  SCAMPI  B2  Noted  Weaknesses 


ment  and  stakeholder  involvement  plans) 
or  the  CM  group.  The  remaining  issues 
were  the  responsibility  of  the  TSP  team. 
Most  issues  were  focused  within  the 
Decision  Analysis  and  Resolution  (DAR) 
and  Causal  Analysis  and  Resolution 
(CAR)  PAs.  The  specifics  of  each  of 
these  are  discussed  in  the  PA  section 
below. 

Based  on  the  results  of  this  initial 
SCAMPI  B,  the  team  continued  its  pro¬ 
ject  work.  The  major  focus  was  on  exe¬ 
cuting  the  team’s  CAR  process  and 
addressing  the  documentation  and 
process  framework  issues.  Significantly, 
the  team  did  not  devote  any  special 
resources  to  the  CMMI  preparatory 
effort.  After  this  finding,  preparatory 
work  was  done  by  the  team  and  led  by 
the  team’s  process  manager  (a  standard 
TSP  role)  as  part  of  normal  work  duties. 
About  four  months  into  this  effort  the 
309th  realized  that  DAR  could  not  be 
solely  addressed  at  the  organizational 
level  and  a  new  process  requirement  for 
DAR  implementation  was  pushed  down 
to  the  project  level.  The  team’s  TSP 
coach  developed  a  draft  process  script 
and  team  training  was  conducted.  No 
opportunity  to  execute  the  DAR  process 
occurred  before  the  second  SCAMPI  B. 

The  weaknesses  and  risks  identified 
by  the  second  SCAMPI  B  are  identified 
in  Table  2.  It  is  important  to  note  that 
the  assessment  team  for  the  second 
SCAMPI  B  was  different  than  the  first 
and  that  this  team  chose  to  identify  areas 
for  improvement  in  the  low-risk  areas, 
whereas  the  first  team  did  not.  These 
new  results  gave  the  GTACS  team  a  dif¬ 
ferent  and  more  thorough  understanding 
of  the  remaining  weaknesses. 

Of  the  weaknesses  noted  there  were 
three  groupings:  DAR  (seven  High 
Artifact,  three  Medium  Artifact,  and  one 
Low  Document);  Organizational  Process 
Performance  (OPP)  (13  High  Non- 
Team,  one  Medium  Non-Team,  and  one 
Low  Non-Team);  and  Training  (one 
High  Artifact,  two  High  Non-Team,  12 
Medium  Document,  one  Low  Artifact, 
and  one  Low  Non-Team).  The  other 
weaknesses  noted  were  scattered 
throughout  the  model.  Of  these,  the 
most  significant  for  the  purposes  of  this 
article  were  the  seven  Medium  Process 
weaknesses.  These  reflected  the  fact  that 
the  team  had  a  process  gap.  In  these 
seven  weaknesses  there  were  three 
process  gaps:  1)  a  lack  of  traceability 
matrices  in  the  team’s  engineering  work 
packages,  2)  a  missing  checklist  item  in 
the  team’s  high-level  design  inspection 
checklist,  and  3)  the  team’s  implementa¬ 


tion  of  statistical  process  control  (SPC) 
to  monitor  selected  subprocesses.  Of  these, 
only  the  SPC  issue  required  a  major 
change  in  the  team’s  practices.  It  is  dis¬ 
cussed  in  detail  below.  The  team’s 
approach  to  requirements  traceability 
had  previously  been  to  include  traceabil¬ 
ity  information  in  the  textual  require¬ 
ments,  design,  and  test  descriptions  and 
to  validate  traceability  via  an  inspection 
checklist  item.  It  was  straightforward  to 
modify  the  engineering  work  package 
template  to  include  the  traceability 
tables.  The  missing  item  in  the  team’s 
high-level  design  inspection  checklist 
was  added,  although  it  had  not  caused 
the  team  issues  in  the  past. 

The  Software  Engineering  Institute 
(SEI)  has  already  performed  a  theoretical 
mapping  of  TSP  to  CMMI  and  deter¬ 
mined  that  DAR  is  partially  addressed  by  the 
TSP,  OPP  is  supported ,  Quantitative 
Process  Management  (QPM)  is  90  percent 
directly  addressed, '  and  CAR  is  60  percent 
directly  addressed  [2].  As  the  GTACS  team 
set  about  to  shore  up  these  weaknesses, 
they  determined  that  these  assessments 
were  generally  accurate;  they  also  came  up 
with  creative  ways  to  update  the  TSP  to 
completely  address  all  of  these  PAs. 

The  PAs 

In  addition  to  the  weaknesses  previously 
described,  there  were  also  minor  weak¬ 
nesses  in  requirements  management,  risk 
management,  and  two  QPM  issues.  Since 
the  initial  preparation  for  DAR  had  been 
only  at  the  group  level,  there  was  no 
DAR  process  or  practice  in  place  for  the 
project.  The  team’s  previous  process 
improvement  discussions,  during  their 
TSP  post-mortems,  had  not  produced 
the  artifacts  necessary  to  meet  CAR 
requirements.  The  TSP  post-mortem 
process  and  PSP  Process  Improvement 
Proposal  (PIP)  process  do  not  require  the 
quantitative  analysis  that  CAR  and  its  link 
to  QPM  does.  The  team  had  not  formal¬ 
ized  its  requirements  management 
process  and  its  documented  risks  man¬ 
agement  process  was  not  consistent  with 
the  TSP  risk  management  process.  The 
QPM  risks  were  labeled  as  medium  risks 
and  related  to  a  lack  of  thresholds  and 
control  limits. 


DAR 

One  of  the  innovations  the  team  came  up 
with  was  in  their  approach  to  the  Level  3 
requirement  for  decision  analysis  and  res¬ 
olution.  Initially,  GTACS  addressed  its 
DAR  requirements  by  adopting  the  orga¬ 
nization’s  DAR  processes  and  forms. 
Organizational  DAR  training  was  held  for 
the  team.  GTACS  created  a  draft  opera¬ 
tional  process  in  the  form  of  a  TSP  script. 
The  DAR  script  was  then  used  by  the 
team  to  analyze  three  different  types  of 
issues:  product  design,  tool  selection,  and 
process.  The  final  DAR  process  was  then 
updated  and  included  in  the  team’s  stan¬ 
dard  process  (see  Figure  1,  next  page). 

The  SEI’s  report  on  TSP  and  CMMI 
identified  all  six  DAR- specific  practices  as 
partially  implemented  and  identified  vari¬ 
ous  launch  meetings  as  points  where  DAR 
activities  are  implemented.  We  believe  this 
partially  implemented  term  underestimates 
the  risk  and  resulting  effort  that  TSP 
teams  will  face  to  meet  DAR  CMMI 
requirements.  A  better  characterization  of 
TSP’s  implementation  of  DAR  is  that  TSP 
is  consistent  with  DAR  philosophy  but  is 
nowhere  near  sufficient.  DAR  is,  at  its 
heart,  a  systems  engineering  sub-process 
for  making  and  documenting  formal  deci¬ 
sions.  In  some  ways  it  is  as  critical  to  the 
systems  engineering  culture  as  inspections 
are  to  software  engineering  or  personal 
reviews  are  to  the  PSP/TSP  approach. 
CMMI  has  elevated  DAR  from  a  practice 
to  a  full-fledged  PA  and  although  TSP  is 
consistent  with  DAR,  TSP  is  insufficient 
to  pass  a  CMMI  assessment.  A  procedure 
like  that  in  Figure  1  is  required  to  produce 
proper  and  meaningful  DAR  artifacts. 

A  TSP  team  must  also  be  trained  in  the 
application  of  DAR.  Based  on  the  back¬ 
ground  of  the  team  members,  this  training 
may  involve  getting  software  engineers  to 
think  like  systems  engineers.  For  the 
GTACS  team,  this  was  surprisingly  diffi¬ 
cult.  While  a  DAR  process,  like  that 
detailed  in  Figure  1,  may  appear  straight¬ 
forward  and  obvious,  software  engineers 
may  question  its  applicability.  For  years  we 
have  observed  good  systems  engineers 
following  processes  like  this  to  make  and 
document  their  systems  designs  and 
design  tradeoffs.  On  the  contrary,  it  has 
been  significantly  more  difficult  to  get 
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DAR  Process  Script 


Purpose 

•  To  guide  the  team  in  making  formal  decisions. 

Entry  Criteria 

Either 

•  A  critical  measurement  exceeds  the  thresholds  defined  in  the  GTACS  DAR  threshold  matrix. 

•  A  critical  decision  needing  a  formal  analysis  is  identified. 

General 

•  Critical  decisions  are  ones  that  have  potential  impact  on  the  project  or  project  team.  Issues  with  multiple  alternative  approaches 
and  multiple  evaluation  criteria  are  particularly  well  suited  for  formal  analysis. 

Tailoring 

•  This  procedure  may  be  used  to  make  and  document  other  decisions. 

Step 

Activities 

Description 

1 

Planning 

-  A  Point  of  Contact  (POC)  is  assigned. 

•  The  POC  may  be  self-assigned  if  the  POC  is  responsible  for  the  critical  decision. 

•  The  team  lead  assigns  the  POC  otherwise. 

-  The  team  that  will  perform  the  DAR  analysis  and  selection  activities  (the  DAR  team)  is  assigned. 

-  The  POC  completes  the  Entry  section  of  the  MXDE  Decision  Analysis  and  Resolution  Coversheet  (section  1). 

-  A  working  directory  is  created  to  hold  the  DAR  artifacts. 

-  An  action  item  is  created  in  the  Project  Notebook  to  track  the  status  of  the  DAR. 

-  The  approval  signatures  required  for  this  DAR  are  determined. 

•  For  DARs  initiated  because  a  critical  measurement  exceeds  the  thresholds  defined  in  the  GTACS  DAR  threshold 
matrix  the  approval  signatures  are  documented  in  the  Stakeholder  Involvement  Plan  (SIP). 

•  For  other  DARs  the  GTACS  Technical  Program  Manager  is  the  approval  authority. 

2 

Identify 

Stakeholders 

-  The  POC  identifies  stakeholders  for  this  DAR  activity.  These  include  the  following: 

•  Those  who  provide  the  alternatives,  risks,  and  historical  data. 

•  The  DAR  team. 

•  Those  who  will  implement  the  decision  the  DAR  results  in. 

3 

Stakeholder 

Input 

-  The  DAR  team  obtains  input  from  the  stakeholders. 

•  Alternative  approaches.  There  is  no  limit  to  the  number  of  alternative  approaches  identified. 

•  Evaluation  Criteria  and  relative  weighting. 

•  Key  risks. 

4 

Evaluation 

Criteria 

-  The  DAR  team  determines  the  evaluation  criteria  and  relative  weighting  after  considering  the  input  from  all  stakeholders. 

-  The  DAR  team  reviews  the  evaluation  criteria  with  the  stakeholders  before  finalizing  the  criteria. 

5 

Selection 

Method 

-  The  DAR  team  determines  the  ranking  and  scoring  method. 

•  Suggested  ranking  and  scoring  methods  are  found  in  the  DAR  Tools  document. 

•  The  DAR  team  must  agree  on  a  scoring  method,  the  scoring  range,  and  have  a  common  understanding  of  what  the 
scores  represent. 

-  The  selected  approach  is  documented  on  the  MXDE  Decision  Analysis  and  Resolution  Coversheet  (section  II). 

6 

Rank  Each 
Approach 

-  For  each  alternative,  the  DAR  team  must  assign  a  score  to  each  decision  criteria,  employing  the  ranking  and  scoring 
method  previously  selected. 

-  The  total  weighted  score  for  each  alternative  is  determined. 

7 

Make  a 

Decision 

-  The  DAR  team  makes  a  decision  and  reviews  it  with  the  stakeholders  making  changes  if  necessary. 

-  The  stakeholders  review  is  captured  on  the  MXDE  Decision  Analysis  and  Resolution  Coversheet  (section  III). 

-  The  final  decision  is  captured  on  the  MXDE  Decision  Analysis  and  Resolution  Coversheet  (section  IV). 

8 

Post-Mortem 

-  The  effort  expended  on  this  DAR  is  captured  on  the  MXDE  Decision  Analysis  and  Resolution  Coversheet  (section  IV). 

-  Approval  signatures  are  obtained  and  recorded  on  the  MXDE  Decision  Analysis  and  Resolution  Coversheet  (section  IV). 

-  DAR  lessons  learned  are  captured  in  the  DAR  notes. 

-  All  DAR  documents  are  captured  and  archived  per  the  GTACS  Data  Management  Plan  (DMP). 

•  The  completed  MXDE  Decision  Analysis  and  Resolution  Coversheet. 

•  Scoring  and  analysis  worksheets. 

•  CM  is  notified  that  the  DAR  is  complete  and  that  the  DAR  artifacts  can  be  archived  to  the  GTACS  data 
management  repository. 

Exit  Criteria 


-  The  MXDE  Decision  Analysis  and  Resolution  cover  sheet  is  completely  filled  out. 

-  The  artifacts  produced  during  the  DAR  activities  have  been  archived  in  accordance  with  the  GTACS  DMP. 


Figure  1:  The  GTACS  Team’s  DAR  Process  Script 


purely  software  engineers  to  document 
their  design  reasoning  with  the  same  rigor. 
It  is,  however,  a  basic  engineering  practice 
that  can  be  easily  learned.  Our  experience 
with  the  GTACS  team  confirmed  this 
observation  that  software  engineers  are 
unfamiliar  with  systems  engineering  tech¬ 
niques  for  formal  decision  making  and 
documentation  but  can  be  easily  trained  to 
use  these  techniques. 

QPM  and  OP P 

One  contentious  area  surrounding  CMMI 
High  Maturity  appraisals  and  organiza¬ 
tions  is  the  definition  and  operationaliza¬ 
tion  of  Maturity  Level  4:  Quantitatively 
Managed.  The  formative  book  on  CMMI: 


Guidelines  for  Process  Integration  and  Product 
Improvement  describes  Maturity  Level  4  as 
the  following  [3]: 

Maturity  Level  4:  Quantitatively 
Managed.  At  maturity  level  4,  the 
organization  and  projects  establish 
quantitative  objectives  for  quality 
and  process  performance  and  use 
them  as  criteria  in  managing 
processes.  Quantitative  objectives 
are  based  on  the  needs  of  the  cus¬ 
tomer,  end  users,  organization,  and 
process  implementers.  Quality  and 
process  performance  is  under¬ 
stood  in  statistical  terms  and  is 
managed  throughout  the  life  of 


the  processes. 

For  selected  subprocesses, 
detailed  measures  of  process  per¬ 
formance  are  collected  and  statisti¬ 
cally  analyzed.  Quality  and  process 
performance  measures  are  incor¬ 
porated  into  the  organization’s 
measurement  repository  to  sup¬ 
port  fact-based  decision  making. 
Special  causes  of  process  variation 
are  identified  and,  where  appropri¬ 
ate,  the  sources  of  special  causes 
are  corrected  to  prevent  future 
occurrences. 

A  critical  distinction  between 
maturity  levels  3  and  4  is  the  pre¬ 
dictability  of  process  performance. 
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Figure  2:  Portion  of  a  Standard  Process  Data  Worksheet  for  the  GTACS  Squadron 


At  maturity  level  4,  the  perfor¬ 
mance  of  processes  is  controlled 
using  statistical  and  other  quantita¬ 
tive  techniques,  and  is  quantitative¬ 
ly  predictable.  At  maturity  level  3, 
processes  are  typically  only  qualita¬ 
tively  predictable. 

Assuming  an  organization  has  achieved 
Maturity  Level  3,  the  concepts  for  Level  4 
are  achieved  by  implementing  the  practices 
and  satisfying  the  goals  for  OPP  and 
QPM.  The  team  weaknesses  identified  at 
Level  4  in  QPM  and  OPP  were  due  to  the 
facts  that  GTACS  data  was  not  analyzed  at 
the  sub-process  level  and  the  data  analyses 
did  not  address  an  understanding  of 
process  variability.  To  understand  the  root 
cause  of  these  issues,  one  must  understand 
how  standard  TSP  projects  use  data  to 
quantitatively  manage  themselves. 

TSP  uses  data  for  three  purposes:  pro¬ 
ject  planning,  project  monitoring  and  over¬ 
sight,  and  process  improvement.  For  pro¬ 
ject  monitoring,  TSP  fundamentally  con¬ 
siders  the  software  development  process 
as  a  single  entity  whose  purpose  is  to  help 
guide  the  production  of  products.  Earned 
Value  (EV),  TSP’s  primary  tool  for  analyz¬ 
ing  schedule  and  cost,  measures  the  whole 
process  and  not  subprocesses.  TSP’s  two 
primary  tools  for  monitoring  quality. 
Percent  Defect  Free  (PDF)  and  Process 
Quality  Index  (PQI)  also  do  not  focus  at 
the  sub-process  level.  PDF  considers  the 
whole  product  and  the  whole  process.  PQI 
focuses  on  the  evolving  quality  of  product 
parts  by  analyzing  the  whole  process  used 
to  produce  them.  Its  usual  use  is  to  identi¬ 
fy  potentially  troublesome  parts  for  addi¬ 
tional  quality  analysis.  In  addition,  none  of 
these  measures  consider  variability  from 
the  statistical  process  control  perspective. 
EV  considers  only  how  actual  cost  and 
schedule  performance  is  varying  from  the 
planned  performance.  Both  PDF  and  PQI 
consider  how  quality  performance  varies 
from  TSP  supplied  benchmarks. 

OPP  looks  at  quantitative  manage¬ 
ment  from  a  top-down  perspective.  After 
the  organization  determines  the  critical 
processes  (or  subprocesses)  and  associat¬ 
ed  measures,  analysis  procedures,  and 
performance  models,  a  project  can  then 
use  the  practices  of  QPM  to  fulfill  pro¬ 
ject  OPP  requirements.  The  organiza¬ 
tion’s  OPP  requirements  define  the  key 
organizational  metrics  as  cost  perfor¬ 
mance  index,  schedule  performance 
index,  yield,  and  rework.  The  team’s  base 
TSP  practices  are  collecting  all  the  mea¬ 
sures  needed  to  meet  these  requirements. 
Figure  2  is  a  portion  of  the  squadron’s 
historical  data  worksheet  showing  the  key 


measures  the  project  must  collect  and 
submit  and  the  key  metrics  derived  from 
those  measures. 

As  noted  earlier,  the  SCAMPI  B 
assessment  team  had  identified  the  team’s 
use  of  EV  and  PQI  (the  team  was  not 
using  the  TSP  PDF  metric  because  it  did 
not  add  value  for  its  work)  as  possibly  not 
fulfilling  the  intent  of  the  variability  of 
subprocesses  clauses  of  QPM.  After  dis¬ 
cussion,  the  team  decided  to  track  rework 
and  the  forecast  completion  date  of  its 
various  work  products.  These  also  sup¬ 
ported  the  team’s  two  highest  priority 
project  goals:  finishing  its  work  on  time 
and  having  low  rework.  The  key  selection 
criteria  for  these  two  metrics  were  that 
they  could  be  tracked  during  the  project, 
that  corrective  action  could  be  taken  if 
they  were  trending  beyond  limits  or  goals, 
and  that  they  were  of  relatively  low  cost 
to  implement. 

The  team’s  EV  tool  computed  the 
forecast  completion  date  of  the  project 
and  because  of  the  way  the  project  plan 
was  set  up,  it  could  also  compute  the 
forecast  completion  date  of  each  of  the 
project  subparts.  The  team  reviewed 
these  forecasts  at  the  subpart  level  every 
week.  Only  once,  when  a  team  member 
had  a  medical  condition  that  required 
unplanned  long-term  leave  did  a  forecast 
fall  past  the  project  end  date,  causing  the 
team  to  replan  its  approach  for  this  par¬ 
ticular  subpart.  This  matches  our  prior 
TSP  experience  where  the  TSP  EV  pro¬ 
ject  tracking  process  leads  the  team  to 
meet  its  schedule  commitments  [4]. 


The  team  was  easily  able  to  use  rework 
in  a  way  that  satisfied  the  CMMI  assessor’s 
need  to  see  the  team  reviewing  process 
variability.  Rework  time  for  this  TSP  team 
was  defined  as  time  recorded  in  the  defect 
logs.  Percentage  rework  was  rework  time 
divided  by  total  task  time.  Good  historical 
data  existed  from  the  team’s  prior  projects. 
Rework  percentage  was  computed  weekly 
and  reviewed  during  the  team’s  weekly 
meeting  for  both  the  project’s  subparts  and 
the  project  as  a  whole.  Rework  remained 
within  control  limits  throughout  the  entire 
project  for  all  project  parts.  Figure  3  (see 
page  20)  is  the  project-level  rework  plot 
that  was  reviewed  by  the  team  during  its 
weekly  meeting.  The  rework  percentage  for 
each  of  the  team’s  subparts  and  the  project 
as  a  whole  were  each  plotted.  The  plots 
each  included  the  subpart  or  project  under 
review,  the  organizational  goal  (10  percent), 
the  Upper  Control  Limit  (10.46  percent), 
and  the  normalized  (to  the  project  sched¬ 
ule)  plots  for  previous  projects. 

The  good  news  is  that  the  data  collec¬ 
tion  required  by  the  TSP  provides  all  and 
more  of  the  data  needed  to  perform  such 
analyses.  Using  these  data,  the  GTACS 
project  was  able  to  come  up  with  QPM 
analyses  that  focused  on  variability  for 
effort,  schedule,  and  quality  performance 
(such  as  rework)  within  predicted  parame¬ 
ters.  The  team  updated  their  weekly  meet¬ 
ing  process  to  address  each  of  these  mea¬ 
sures,  to  see  if  they  were  in  control,  and  to 
bring  items  that  had  gone  astray  back 
under  control.  GTACS  also  added  items 
to  the  TSP  post-mortem  process  to  collect 
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project  closeout  data  that  could  be  used  to 
determine  process  performance  and  vari¬ 
ability  overall  and  at  the  sub-process  lev¬ 
els.  These  data  were  then  standardized  for 
sharing  across  the  organization,  support¬ 
ing  the  requirements  of  OPP  (Figure  3). 

CAR 

The  TSP  process  as  it  currently  stands  calls 
for  a  detailed  post-mortem  analysis  of 
project  and  process  data,  including  identi¬ 
fication  of  improvements.  This  provides  a 
great  deal  of  support  for  the  Level  5  CAR 
requirement;  however,  the  TSP  does  lack 
CAR  formality  and  feedback  to  determine 
if  implemented  process  improvements 
really  worked.  In  order  to  shore  up  these 
issues,  the  GTACS  team  updated  the  post¬ 
mortem  script  to  directly  address  CAR. 
They  created  a  requirement  for  a  CAR 
report ,  which  formally  douments  the  TSP 
post-mortem  by  capturing  the  data  analy¬ 
ses  performed,  weaknesses  identified,  and 
the  suggested  process  changes  to  address 
these  weaknesses.  The  report  also  adds  to 
the  TSP  post-mortem  an  analysis  of  the 
impact  of  previous  process  improvements. 

Training 

The  TSP  rollout  strategy  that  the  GTACS 
team  used  included  PSP  training  for  all 
engineers  and  managing  TSP  teams  train¬ 
ing  for  the  team  lead  and  the  GTACS 
TPM.  This  approach  provided  the  primary 
training  for  eight  of  the  21  PAs.  Additional 
organizational  specific  training  on  policy 
was  still  required.  The  PAs  addressed  by 
PSP /TSP  were  project  planning,  project 
monitoring  and  control,  integrated  project 
management,  integrated  teaming,  process 


and  product  quality  assurance,  measure¬ 
ment  and  analysis,  and  CAR.  Verification 
was  partially  addressed.  Training  was 
required  for  the  management  PAs  of  risk 
management  and  quantitative  project  man¬ 
agement,  all  the  engineering  PAs  (Require¬ 
ments  Development,  Requirements  Man¬ 
agement,  Technical  Solution,  Product  Inte¬ 
gration,  Validation,  and  Verification),  the 
support  PAs,  configuration  management, 
and  DAR,  and  all  the  process  management 
PAs  (Organizational  Process  Focus,  Org¬ 
anizational  Process  Definition,  Organiza¬ 
tional  Process  Performance,  and  Organ¬ 
izational  Innovation  and  Deployment). 

The  team  addressed  the  training  issue 
by  creating  a  team  training  plan  that  dis¬ 
cussed  how  new  team  members  acquired 
the  skills  needed  to  become  full  team 
members.  This  included  an  approach  to 
obtaining  GTACS  domain  knowledge,  the 
tools  and  technologies  used  by  the  team, 
the  processes  used  by  the  team,  and  the 
key  organizational  training  needed  to  sup¬ 
port  the  team.  Most  of  the  details  of  these 
training  packages  had  been  in  existence 
for  several  years  but  were  not  structured 
and  organized.  In  fact,  the  team  had  a 
longstanding  improvement  proposal  to 
organize  its  training  approach. 

Summary 

The  GTACS  team  in  309th  SMXG  at  Hill 
Air  Force  Base,  Utah,  successfully  used  the 
TSP  in  reaching  their  goal  of  CMMI  Level 

5.  In  order  to  do  so,  they  adapted  from  and 
added  to  the  TSP  scripts,  measures,  and 
forms  in  ways  that  they  believe  can  help 
other  TSP  teams  also  achieve  this  feat,  as  far 
as  can  be  done  by  a  single  focus  project.^ 


Related  Literature 

The  topic  of  relating  TSP  practice  to 
CMM-based  assessments  has  been 
addressed  in  two  thought  papers  and  at 
least  two  case  studies.  The  thought  papers 
studied  the  problem  in  the  abstract  by 
comparing  a  theoretical  TSP  project 
against  a  model.  Davis  and  McHale  [5] 
first  compared  TSP  against  the  CMM  and 
concluded  that  TSP  implements  a  majority  of 
the  key  practices  of  the  SW-CMM.  McHale 
and  Wall  [2]  later  extended  this  study  to 
the  CMMI.  They  concluded,  that  TSP  can 
instantiate  a  majority  of  the  project-oriented  spe¬ 
cific  practices  of  CMMI. 

Naval  Air  Systems  Command  used 
TSP  to  advance  their  CMM  efforts.  Their 
experience  report  compared  their 
approach  of  using  TSP  to  implement 
CMM  improvement  versus  non  TSP 
based  CMM  improvement  approaches. 
They  reported  that  they  halved  the  time 
needed  to  move  from  CMM  level  1  to 
CMM  level  4  by  basing  their  process  on 
TSP  [6,  7] .  Cedillo  reported  that  TSP  actu¬ 
ally  accelerates  CMM/  CMMI  implementation 
in  a  small  setting  where  the  process 
improvement  approach  of  a  small  startup 
company  was  based  on  TSP  [8]. 
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Figure  3:  Variability  in  Rework  as  Tracked  by  the  GTACS  Team 
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